GH-49102: [CI] Add type checking infrastructure and CI workflow for type annotations by rok · Pull Request #48618 · apache/arrow

rok · 2025-12-22T00:15:41Z

Rationale for this change

This is the first in series of PRs adding type annotations to pyarrow and resolving #32609.

What changes are included in this PR?

This PR establishes infrastructure for type checking:

Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows
Configures type checkers to validate stub files (excluding source files for now)
Adds PEP 561 py.typed marker to enable type checking
Updates wheel build scripts to include stub files in distributions
Creates initial minimal stub directory structure
Updates developer documentation with type checking workflow

Are these changes tested?

No. This is mostly a CI change.

Are there any user-facing changes?

This does not add any actual annotations (only py.typed marker) so user should not be affected.

GitHub Issue: [Python] Type checking support #32609
GitHub Issue: [CI] Add type checking infrastructure and CI workflow for type annotations #49102

AlenkaF

Thank you @rok for working on this! Splitting up the main PR sounds like a great idea to me.

I only have one small question, otherwise the changes look good to me.

ci/scripts/python_test_type_annotations.sh

rok · 2025-12-23T10:26:37Z

Thanks for review @AlenkaF ! I think we probably want @raulcd to look at this too before merging?

AlenkaF · 2025-12-23T10:32:04Z

Yes, let's wait for Raul 👍

raulcd

Thanks for the PR @rok sorry it took me a while to review with holidays, release, etcetera

ci/scripts/python_test_type_annotations.bat

ci/scripts/python_test_type_annotations.sh

docs/source/developers/python/development.rst

ci/scripts/python_wheel_validate_contents.py

ci/scripts/python_wheel_xlinux_build.sh

raulcd · 2026-01-13T19:27:53Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-13T19:30:08Z

Revision: 28fba3a

Submitted crossbow builds: ursacomputing/crossbow @ actions-8a459250fe

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T10:41:02Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


@raulcd do you think we actually want to be able to toggle these? I'm thinking we don't really need them

I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

Ok, agreed with your reasoning. I also introduced PYARROW_TEST_ANNOTATIONS to the windows script.

rok · 2026-01-14T10:41:33Z

Thanks for review @raulcd ! I've responded to your points.

raulcd · 2026-01-14T11:41:13Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

raulcd · 2026-01-14T11:42:42Z

ci/scripts/python_wheel_macos_build.sh

+# We first populate stub docstrings and then build the wheel
+python setup.py build_ext --inplace
+python -m pip install griffe libcst
+python ../dev/update_stub_docstrings.py pyarrow-stubs


this is not necessary, right?

Same as #48618 (comment). It's a noop and we don't have to introduce python logic in stub PRs if we do here.

ci/scripts/python_wheel_validate_contents.py

raulcd · 2026-01-14T11:44:04Z

ci/scripts/python_wheel_windows_build.bat

+@REM We first populate stub docstrings and then build the wheel
+%PYTHON_CMD% setup.py build_ext --inplace
+%PYTHON_CMD% -m pip install griffe libcst


Until we have real annotations this isn't necessary, right?

I added running dev\update_stub_docstrings.py, so I suppose it's a noop now and we can keep it to make sure it doesn't fail before introducing actual stubs? This will keep CI logic out of stub PRs.

…g check

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

rok · 2026-02-05T10:18:01Z

I rebased on main to fix the docs CI issues.
I'd like to continue on the annotation PRs next week so I'll merge Sunday evening.

rok requested review from AlenkaF, assignUser, jonkeane, kou and raulcd as code owners December 22, 2025 00:15

github-actions bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Dec 22, 2025

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from afe4699 to 30017ff Compare December 22, 2025 00:20

This was referenced Dec 22, 2025

GH-32609: [Python] Add type annotations to PyArrow #47609

Open

GH-49103: [Python] Add internal type system stubs (_types, error, _stubs_typing) #48622

Draft

AlenkaF approved these changes Dec 23, 2025

View reviewed changes

ci/scripts/python_test_type_annotations.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 23, 2025

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 23, 2025

raulcd reviewed Jan 13, 2026

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 14, 2026

rok commented Jan 14, 2026

View reviewed changes

raulcd reviewed Jan 14, 2026

View reviewed changes

rok and others added 24 commits February 5, 2026 11:15

Use PYARROW_TEST_ANNOTATIONS in windows build, disable wheel docstrin…

2209d00

…g check

work on dev/update_stub_docstrings.py

e542a00

further work on dev/update_stub_docstrings.py

c80b711

lint

4ef1177

add click for docstring population

4ac5626

Remove dependencies on click and griffe

4e75cf6

fix import paths

63dbf61

add PYARROW_TEST_ANNOTATIONS to AMD64 Windows 2022 Python 3.13

e1fe00d

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

move check to pre-commit

40feef7

change pre-commit, add note

b528499

change pre-commit script

0136377

Revert from pre-commit to ci/scripts/python_test_type_annotations.sh

aa4d372

Apply suggestions from code review

e81be9d

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

apply review suggestion

0cbb6d0

fix shellcheck

7885170

Try single build

ded10e9

lint

6579023

review feedback

6dd417a

reintroduce pythonVersion = 3.10

1311583

revert change

16de131

post rebase fix

883ac54

--no-build-isolation bypasses pyproject.toml's build requirements

a41b6fe

libcs added

7102d94

Added DYLD_LIBRARY_PATH export to python_build.sh

02c83e0

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from a88b174 to 02c83e0 Compare February 5, 2026 10:16

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 5, 2026

rok requested review from AlenkaF and pitrou February 5, 2026 10:18

Conversation

rok commented Dec 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Dec 23, 2025

Uh oh!

AlenkaF commented Dec 23, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rok commented Dec 22, 2025 •

edited by github-actions bot

Loading